Skip to content

Implement migrate to Bluesky#371

Merged
micahflee merged 58 commits intomainfrom
369-migrate-to-bluesky
Feb 20, 2025
Merged

Implement migrate to Bluesky#371
micahflee merged 58 commits intomainfrom
369-migrate-to-bluesky

Conversation

@micahflee
Copy link
Member

@micahflee micahflee commented Jan 20, 2025

Resolves #369.

  • Updates user interface for X accounts to add navigation to the right sidebar, including a new "Migrate to Bluesky" page
  • Allow the user to connect a Bluesky account to their X account in Cyd using OAuth (and to disconnect the Bluesky account), including using cyd:// protocol links
  • Add a join table migration in the X account SQLite database to track which tweets have been migrated and where
  • Support basic migrating of tweets
  • Support migrating complex tweets (blocked by When indexing tweets, save media, reply status, and quote status too #354):
    • Tweets with media
    • Quote tweets
    • Threads of tweets
  • When building the X archive, link to Bluesky posts that have been migrated
  • Make this a premium features -- and make sure the premium flow works smoothly
  • Tell people they need to archive their tweets before they can migrate them
  • Detect if an archive/import happened before this release, and if so tell people to import again first

Before reviewing this PR, we should review and merge #393. I've merged those commits into this branch, so merging that into main will make this branch have far fewer commits.

Here's a brief tour of this feature and the related changes in this branch. First, I'm going to start with a fresh instance of Cyd Dev by deleting the app data and deleting the data in my test account:

rm -r ~/Library/Application\ Support/Cyd\ Dev/
rm -r ~/Documents/Cyd\ Dev/X/nexamind91326/

New navigation

I open Cyd and add my test X account and login to X. The interface looks like this:

Screenshot 2025-02-18 at 4 15 14 PM

In the right sidebar of an X account, there's now nav menu with "Local Database", "Delete from X", and "Migrate to Bluesky". Before there was no nav, and you just got around via buttons in the wizard.

As part of this PR, I've also changed the messaging it shows you when you haven't imported any data yet. If you switch to Delete from X, it shows this prominent warning and encourages you to build your local database first:

Screenshot 2025-02-18 at 4 17 39 PM

A similar warning was shown before, but I think this looks better. Also on the Delete from X page, it shows the disabled options (delete my tweets, delete my retweets, delete my looks) isn't of completely hiding them.

The same warning is on the Migrate to Bluesky page:

Screenshot 2025-02-18 at 4 18 37 PM

Info/warning alerts after you have some data

Next, I built a database from scratch, and now the three pages show that last time the database was built (or imported from an archive). The Local Database page doesn't have a button:

Screenshot 2025-02-18 at 4 22 04 PM

But the Delete from X and Migrate to Bluesky pages do:

Screenshot 2025-02-18 at 4 22 49 PM

Note that it determines whether or not you have data by loading the lastFinishedJob_indexTweets, lastFinishedJob_indexLikes, and lastFinishedJob_importArchive account config settings, which store timestamps. If I go through and update the timestamps in the SQL database to be before February 18, 2025, the Migrate to Bluesky page shows you this warning:

Screenshot 2025-02-18 at 4 24 52 PM

REMINDER: Before releasing this in prod, we should update this date to be the release date. It's hardcoded in XWizardMigrateBluesky.vue.

The Cyd archive before migrating

At this point, if you browse the Cyd archive, it looks like this. It has all of the fancy stuff from #393, and there's a link to the original tweet, but there's no link to the migrated version in Bluesky yet.

Screenshot 2025-02-18 at 4 26 55 PM

Bluesky OAuth

I put my test Bluesky handle (nexamind-cyd.bsky.social) in Cyd and click Connect:

This opens a new tab in my browser to authenticate to Bluesky:

Screenshot 2025-02-18 at 4 28 57 PM

Screenshot 2025-02-18 at 4 29 11 PM

I click Accept, and it redirects to a special cyd.social URL:

Screenshot 2025-02-18 at 4 29 44 PM

I click "Open Link" and the Cyd app responds saying I'm successfully connected (note you can test canceling in the OAuth flow too, everything should be handled). Once you're connected, you can start migrating to Bluesky. It tells you the number of tweets it can migrate, the number it can't migrate because they're replies, and the number that have already been migrated:

Screenshot 2025-02-18 at 4 30 09 PM

When you click "Start Migrating to Bluesky", it verifies that you have premium access.

Premium check

This is a premium feature, so I want to make the premium flow is seamless. I had to update how the premium check works a bit from before. But basically, if you're not signed in to a Cyd account or you're signed in but haven't paid for premium, it brings you to premium check page and shows you the feature is "Migrate tweets to Bluesky":

Screenshot 2025-02-18 at 4 33 29 PM

The old premium check view used to determine exactly what features were required by loading account settings from the database (like, deleteLikes). But this is a new premium feature that isn't storing anything in the account settings, so I've introduced two new localStorage items: premiumCheckReason-${accountID} and premiumTasks-${accountID}.

If a user doesn't have premium, before redirecting to the premium check, it sets premiumCheckReason to migrateTweetsToBluesky and it sets premiumTasks to a list of tasks, which is just one right now ("Migrate tweets to Bluesky"):

    // Premium check
    if (await window.electron.getMode() == "open") {
        if (!await showQuestionOpenModePremiumFeature()) {
            return;
        }
    }
    // Otherwise, make sure the user is authenticated
    else {
        if (!props.userAuthenticated || !props.userPremium) {
            localStorage.setItem(`premiumCheckReason-${props.model.account.id}`, 'migrateTweetsToBluesky');
            localStorage.setItem(`premiumTasks-${props.model.account.id}`, JSON.stringify(['Migrate tweets to Bluesky']));
            emit('setState', XState.WizardCheckPremium);
            return;
        }
    }

When you're trying to delete your data, before redirecting the user to premium check, it now sets premiumCheckReason in localStorage to deleteData. And the premium check view uses this information to determine what features to display, and also what the back button should say and where it should go. So if you're trying to delete data with a bunch of premium features enabled, the premium check looks like this now, where the back button brings you to the delete review page:

Screenshot 2025-02-18 at 4 39 21 PM

Anyway, back to migrating to Bluesky: I envision people may want to pay for premium directly so they can move forward and migrate their tweets, so I want this flow to be flawless. After logging in (my account has already paid for premium) it shows this:

Screenshot 2025-02-18 at 4 40 51 PM

The button is "Migrate to Bluesky" in this case, but if I were doing this flow from the Delete from X page, then the button would bring me back to deletion options.

Once I'm signed in and I have premium, clicking "Start Migrating to Bluesky" will actually migrate to Bluesky.

Migration

As soon as you click the button, it migrates all of your tweets super fast. It gets all 58 of mine in a few seconds. Here's a screenshot I snagged.

Screenshot 2025-02-18 at 4 42 20 PM

When it's done, it tells you how many tweets it migrated and there's a link to load your Bluesky profile:

Screenshot 2025-02-18 at 4 42 51 PM

If there are any errors while migrating, it shows you the errors here too, which I'll get to in a second.

Over on Bluesky, everything should get migrated. Here's a video that was migrated:

Screenshot 2025-02-18 at 4 48 15 PM

Here's a tweet with multiple images and also a thread:

Screenshot 2025-02-18 at 4 48 39 PM

Here's a quoted tweet, quoting a a different X account:

Screenshot 2025-02-18 at 4 54 39 PM

The migrate to Bluesky feature supports the following:

  • Posts are backdated
  • The text of the tweet is updated to replace t.co links with actual URLs, and it adds facets to turn these URLs into actual links
  • It handles self-replies, which keeps twitter threads intact when migrating them
  • It handles self-quotes, and if you quote tweet someone else, it adds a website card
  • It handles images and video
  • It handles recordWithMedia -- which is a quote post that also includes media

After you migrate your tweets to Bluesky, there's now a button to delete them if you want (I used this for debugging):

Screenshot 2025-02-18 at 4 57 06 PM

That should delete all of the posts that were migrated. And when you browse your archive, all of the migrated tweets should now have a "migrated to Bluesky" link which links to the new Bluesky post.

Screenshot 2025-02-18 at 4 57 38 PM

@micahflee
Copy link
Member Author

I've got atproto OAuth working!

Screen.Recording.2025-02-04.at.3.08.26.PM.mov

@micahflee micahflee marked this pull request as ready for review February 19, 2025 00:59
@micahflee micahflee removed the blocked label Feb 19, 2025
@redshiftzero redshiftzero self-requested a review February 19, 2025 17:08
@redshiftzero
Copy link
Contributor

redshiftzero commented Feb 19, 2025

Two snags testing this so far:

  1. Not a bug but posting here for posterity: After clicking "Open Link" I needed to specify the "Cyd Dev" application for it to open in there, else it opened in a new Electron window.
  2. I'm getting this error when I try to auth to Bluesky:
Screenshot 2025-02-19 at 1 22 38 PM

Upon inspection, my local user table is empty, and the blueskyMigrateTweet method added in this PR requires at least one entry

@micahflee
Copy link
Member Author

2. I'm getting this error when I try to auth to Bluesky:
Screenshot 2025-02-19 at 1 22 38 PM

Upon inspection, my local user table is empty, and the blueskyMigrateTweet method added in this PR requires at least one entry

Before, the xAccount table stored username but not userID. I fixed it so that it now loads userID, so the Bluesky migration code can pull the current user's ID from the account row, and no longer has to use the user table.

In order to fix this, I also added several commits that I think actually improve the reliability of X platform support in general. In XViewModel.login in the renderer, it used to jump through hoops to figure out the logged in user's username. It would load the homepage, click on the profile button, then pull the username from the URL. It would also click on the profile image and grab the profile picture from the DOM.

Now, it's much simpler. It loads the homepage and then uses javascript to make an HTTP request to https://api.x.com/graphql/WBT8ommFCSHiy3z2_4k1Vg/Viewer?variables=%7B%22withCommunitiesMemberships%22%3Atrue%7D&features=%7B%22profile_label_improvements_pcf_label_in_post_enabled%22%3Atrue%2C%22rweb_tipjar_consumption_enabled%22%3Atrue%2C%22responsive_web_graphql_exclude_directive_enabled%22%3Atrue%2C%22verified_phone_label_enabled%22%3Afalse%2C%22creator_subscriptions_tweet_preview_api_enabled%22%3Atrue%2C%22responsive_web_graphql_skip_user_profile_image_extensions_enabled%22%3Afalse%2C%22responsive_web_graphql_timeline_navigation_enabled%22%3Atrue%7D&fieldToggles=%7B%22isDelegate%22%3Afalse%2C%22withAuxiliaryUserLabels%22%3Afalse%7D. This is a JSON object that includes all of the information we want: username, user ID, profile picture URL, and also follower count, following count, number of tweets, and number of likes.

The other code that makes manual graphQL requests (for deleting tweets, RTs, and likes) hit endpoints at https://x.com/graphql/, but this code now also hits an endpoint at https://api.x.com/graphql/, which is a different domain. In order to handle the CSRF token stuff, I refactored how cookies in XAccountController work so that it logs all cookies for all domains, not just for x.com, and when you get a cookie you choose the domain you want.

There was also a totally different hacky way of determining the current user's number of tweets and likes. I ripped all of that code out in favor of just running the login function again, to get that info from this one API request.

Now that all of that is refactored, in the blueskyGetTweetCounts IPC function, I get the user ID from this.account.userID.

Copy link
Contributor

@redshiftzero redshiftzero left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

getting the user ID from this.account.userID resolved for me! LGTM

@micahflee micahflee merged commit 6803e3e into main Feb 20, 2025
1 check passed
@micahflee micahflee deleted the 369-migrate-to-bluesky branch February 20, 2025 02:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add migrate to Bluesky as a feature of the X platform

2 participants